Verde & Xbatcher -> Any connections / shared use?

[ThomasMGeo]

Verde (https://www.fatiando.org/verde/latest/#) has some spatial splitting and shuffling functions. Does this (could?) have some overlap with Xbatcher?

One example here: https://www.fatiando.org/verde/latest/api/generated/verde.BlockShuffleSplit.html#verde.BlockShuffleSplit

Just thinking out-loud as I dig into the repo's.

Best,
-TM

[maxrjones]

👋 @ThomasMGeo, thanks for opening this issue!

Verde's utility for splitting data definitely overlaps with the development roadmap for xbatcher. It will be useful to consider Verde’s implementation when building out xbatcher's API. A couple key differences are that xbatcher’s splitting and shuffling utilities will operate on batches and xarray datasets or dataarrays whereas verde expects the data array and coordinates to be separated. I think we’ll also want to support both selective and random sampling.

[weiji14]

Just on the sampling strategy, there's been some literature on how typical random train/validation/test splits can lead to overly optimistic model performance estimates (i.e. bad) because it doesn't account for spatial autocorrelation. See this twitter thread https://twitter.com/TejaKattenborn/status/1550416950341271552, and also:

• Kattenborn, T., Schiefer, F., Frey, J., Feilhauer, H., Mahecha, M. D., & Dormann, C. F. (2022). Spatially autocorrelated training and validation samples inflate performance assessment of convolutional neural networks. ISPRS Open Journal of Photogrammetry and Remote Sensing, 5, 100018. https://doi.org/10.1016/j.ophoto.2022.100018
• Roberts, D. R., Bahn, V., Ciuti, S., Boyce, M. S., Elith, J., Guillera-Arroita, G., Hauenstein, S., Lahoz-Monfort, J. J., Schröder, B., Thuiller, W., Warton, D. I., Wintle, B. A., Hartig, F., & Dormann, C. F. (2017). Cross-validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure. Ecography, 40(8), 913–929. https://doi.org/10.1111/ecog.02881
• Valavi, R., Elith, J., Lahoz‐Monfort, J. J., & Guillera‐Arroita, G. (2019). block CV: An r package for generating spatially or environmentally separated folds for k ‐fold cross‐validation of species distribution models. Methods in Ecology and Evolution, 10(2), 225–232. https://doi.org/10.1111/2041-210X.13107

So I would highly recommend going for a block-based sampling approach as with Verde and/or a diagonal sampling (see figure below, middle plot).

[ThomasMGeo]

Yes! Fully agree.

Can make the block sizes large to somewhat get around this with a random strategy, but the other examples shown also look great.

[maxrjones]

I'm looking forward to continuing conversations about the overlap between xbatcher and other libraries like Verde. I just enabled GitHub discussions and transferred this issue over to enable both these types of conversations and usage-focused questions about xbatcher.

[weiji14]

Found out that eo-learn also has some similar spatial sampling functionality. See e.g.:

• https://eo-learn.readthedocs.io/en/latest/reference/eolearn.ml_tools.sampling.html#eolearn.ml_tools.sampling.BlockSamplingTask
• https://github.com/sentinel-hub/eo-learn/blob/v1.2.1/ml_tools/eolearn/ml_tools/sampling.py#L295-L362

They also do train/test split:

• https://eo-learn.readthedocs.io/en/latest/reference/eolearn.ml_tools.train_test_split.html
• https://github.com/sentinel-hub/eo-learn/blob/v1.2.1/ml_tools/eolearn/ml_tools/train_test_split.py#L28-L128

It's a slightly different paradigm since they operate on (numpy) EOPatch objects (though they do have an EOPatch to xarray.Dataset converter at https://eo-learn.readthedocs.io/en/latest/reference/eolearn.visualization.extra.xarray.html#eolearn.visualization.extra.xarray.eopatch_to_dataset), while xbatcher is in xarray-land. Still, the code might be a good reference when we work on reinventing the wheel 😆

[weiji14]

Actually, I just realized that eo-learn-ml-tools can somewhat be standalone (see https://github.com/sentinel-hub/eo-learn/tree/v1.2.1/ml_tools), though it still relies on eo-learn-core. Maybe we could just delegate the sampling to eo-learn-ml-tools and focus on other things in xbatcher? Or we can contribute upstream to eo-learn-ml-tools's sampling functionality, e.g. extend it to be compatible with xarray. Might be worth a bit more investigation before we decide to reinvent the wheel.