redis fsspec cache

[mpiannucci]

Hello! I built a simple prototype fs cache using redis as a proof of concept: https://github.com/mpiannucci/redis-fsspec-cache

The reason for building this is that we are running xpublish which does not support async for io bound tasks. This means to scale out, we have to add more workers horizontally. Using redis as our fsspec cache allows us to scale up while sharing recently loaded chunks or blocks close to the compute to minimize latency.

I am not sure if there are implementation details I am missing to make this perfect but it is working well for me so I wanted to share!

[martindurant]

That's great, thank you for sharing. You would be welcome to propose this for inclusion in fsspec or as a separate package in the fsspec org (or not), whatever you feel right.

There has been long standing intent to make file/block caching in fsspec better and more flexible, and this show a nice example of the possibilities. It is a pity, for example, that you needed to write both a filesystem interface and the cacher class too (although I suppose it's possible to use only the latter). Actually, a general-purpose redisFS would be pretty useful too.

I could not immediately tell, is the implementation here supposed to allow for async or batch requests?

xpublish which does not support async for io bound tasks

Isn't this surprising? While the work here may be very useful for lots of people, I would expect xpublish, which is all REST API, to work well for async IO.

[mpiannucci]

I could not immediately tell, is the implementation here supposed to allow for async or batch requests?

I would like this to work for async tasks but I have not tested as such. The cache methods are definitely sync and i couldnt tell if there was a way to make the cache impl methods async? Is that possible? Certainly the redis ios should be non-blocking if possible.

xpublish which does not support async for io bound tasks

Isn't this surprising? While the work here may be very useful for lots of people, I would expect xpublish, which is all REST API, to work well for async IO.

I think I can explain this better... Xpublish itself supports building async routes without issue, but Xarray itself does not support async methods except for when a distributed dask array is being used. I am not using dask distrbuted, so all the xarray functionality behind xpublish is sync. This means that all xarray functionality is blocking so this cache allows for having a whole bunch of blocking workers with a shared cache instead of only a few non blocking async workers. Does that help explain a bit?

[mpiannucci]

Specifically for kerchunked datasets (netcdf or grib2 typically)

[martindurant]

In the case of kerchunk, you have a choice of whether you store (parts of) the original files, or the chunks themselves. I think the latter probably makes much more sense, and that also implies that the code will not be using the file-like API, but cat/cat_ranges.

That brings to the question of async: you would want to do what referenceFS does, and implement only async methods, which can call sync or async methods on the target FS. If that target FS is only ever referenceFS, then you know it will always be async.

[mpiannucci]

Yeah so one of the implementations I included is using the file interface but the chunks (byte ranges) are specifically cached instead of logical blocks. This works well but it is sync.

That brings to the question of async: you would want to do what referenceFS does, and implement only async methods, which can call sync or async methods on the target FS. If that target FS is only ever referenceFS, then you know it will always be async.

okay that is very helpful. I will read referenceFS impl a bit deeper and try to build a simplified async wrapped cacheing impl. Thank you so much for the help

[martindurant]

zarr already has a redis mapper (i.e., all your data is already on redis in zarr format) and a LRU mapper (in memory) but not a combination. In any case, both the cacher for file chunks and a wholefile caching FS would I think be useful to fsspec generally.

[mpiannucci]

OKay so further testing and read_block is very fast with the cache I wrote but cat_file is much slower. So thanks for hte input on the async file system cuz I think thats the route I will go next