CachingFileSystem with http vomits on __del__ under fsspec 2022.10.0

[jwodder]

As of the release of fsspec 2022.10.0 yesterday, the following code:

import fsspec
from fsspec.implementations.cached import CachingFileSystem

fs = CachingFileSystem(
    fs=fsspec.filesystem("http"),
    cache_storage="cache01",
)

with fs.open("https://httpbin.org/uuid", "r", encoding="utf-8") as f:
    print(f.read())

outputs the following traceback on termination:

Traceback (most recent call last):
  File "/Users/jwodder/work/dev/tmp/fsspec/fsspec-cache01a.py", line 10, in <module>
    with fs.open("https://httpbin.org/uuid", "r", encoding="utf-8") as f:
  File "/Users/jwodder/.local/virtualenvwrapper/venvs/tmp-aaf3a4de95a14b1/lib/python3.10/site-packages/fsspec/implementations/cached.py", line 344, in <lambda>
    f.close = lambda: self.close_and_update(f, close)
  File "/Users/jwodder/.local/virtualenvwrapper/venvs/tmp-aaf3a4de95a14b1/lib/python3.10/site-packages/fsspec/implementations/cached.py", line 406, in <lambda>
    return lambda *args, **kw: getattr(type(self), item).__get__(self)(
  File "/Users/jwodder/.local/virtualenvwrapper/venvs/tmp-aaf3a4de95a14b1/lib/python3.10/site-packages/fsspec/implementations/cached.py", line 357, in close_and_update
    c = self.cached_files[-1][path]
KeyError: '/uuid'
Exception ignored in: <function AbstractBufferedFile.__del__ at 0x10ddf55a0>
Traceback (most recent call last):
  File "/Users/jwodder/.local/virtualenvwrapper/venvs/tmp-aaf3a4de95a14b1/lib/python3.10/site-packages/fsspec/spec.py", line 1751, in __del__
  File "/Users/jwodder/.local/virtualenvwrapper/venvs/tmp-aaf3a4de95a14b1/lib/python3.10/site-packages/fsspec/implementations/cached.py", line 344, in <lambda>
  File "/Users/jwodder/.local/virtualenvwrapper/venvs/tmp-aaf3a4de95a14b1/lib/python3.10/site-packages/fsspec/implementations/cached.py", line 406, in <lambda>
  File "/Users/jwodder/.local/virtualenvwrapper/venvs/tmp-aaf3a4de95a14b1/lib/python3.10/site-packages/fsspec/implementations/cached.py", line 357, in close_and_update
KeyError: '/uuid'

Inserting a print statement in WholeFileCacheFileSystem.close_and_update() reveals that the keys in self.cached_files[-1] are URLs (i.e., 'https://httpbin.org/uuid') rather than the /uuid paths that the code expects.

[martindurant]

This appears required but not sufficient to fix the problem:

--- a/fsspec/implementations/http.py
+++ b/fsspec/implementations/http.py
@@ -133,7 +133,7 @@ class HTTPFileSystem(AsyncFileSystem):
     @classmethod
     def _strip_protocol(cls, path):
         """For HTTP, we always want to keep the full URL"""
-        return path
+        return str(path)

(because paths are now URL class, not simple string; this should maybe be fixed on the .path attribute of the file instance)