Is there a way to keep track of cache hits vs actual bytes requested?

[betolink]

fsspec logs the range requests it does but I don't think it makes a distinction of how many bytes came from the cache vs actual requested bytes, would this be too complicated to implement in the caching.py classes?

[betolink]

FWIW I'll try to take a stab at this next week

[martindurant]

Yes, great idea! Everything is known during _fetch, but each cacher has a different implementation of that. Of course, this only applies to files based on AbstractBufferedFile. Also, should clarify that logging messages (e.g., from "s3fs") stating a range was fetched should be requests to the real backend; whereas AbstractBufferedFile.read() reports what the user-facing API saw.

[betolink]

Interesting, I was/am using this

filesystem_spec/fsspec/spec.py

Line 1842 in 2a8e0ee

logger.debug("%s read: %i - %i", self, self.loc, self.loc + length)

which is logged before the actual call to self.cache._fetch(self.loc, self.loc + length), is this the user-facing side of things?

[martindurant]

Oh OK, so we know the request there too - but we don't know at that point the state of the cache (which has several implementations).

[betolink]

Hi @martindurant, I did a first pass and seems to be working as expected, by adding counters in the base class and updating the debug message accordingly I'm now getting these logs:

<File-like object S3FileSystem, file.h5> read: 432000000 - 440000000, total: 276824064, cache hits: 34, cache misses: 33

total is the total requested bytes, and cache hits vs misses in blockcache is a bit tricky because unless we keep reading from the same block over and over we'll have 1 miss + 1 hits on each pass. I added some unit tests for these numbers.

https://github.com/betolink/filesystem_spec/blob/master/fsspec/tests/test_caches.py

I also consolidated the __repr__ and I feel like there is room for some fancy rich representation where we see a line representing the file and the cached sections using a different color and some stats.

Take a look and let me know if this could be a PR.