Add stdlib-compatible shim layer.

[asford]

@martindurant This is a trial-balloon issue for this idea, mostly to see if fsspec is the "right" place for this feature and/or if I've missed an existing implementation. If so, I've a working proof-of-concept built off an older version of dask that could serve as a starting-point PR.

fsspec is a great module for new development or projects built on the existing dask ecosystem and enables an amazing S3-is-my-filesystem paradigm. However, the vast majority of projects make use of the python's built-in file operations, and are tightly coupled to a local filesystem. This causes major development friction when integrating existing third-party libraries into a project, as one almost invariably needs to work out an integration-specific flow between local files and remote storage.

One can workaround this problem with FUSE-based mounts, however this complexifies deployment and containerization. Alternatively, any of several (fsspec, smart_open, pyfilesystem2, et. al.) filesystem abstractions could be used, but updating a third-party component to an alternative filesystem interface is a painful and risky development lift. One either needs to maintain a private fork 😳 or open a massive and risky PR 😬.

I've found that a solid majority of filesystem use cases are covered by a relatively small set of operations, all of which is already covered by fsspec. By providing strictly-compatible shims for a small set of the stdlib (eg: open, os.path.exists, os.remove, glob.glob, shutil.rmtree, et. al.) and then swapping these via import level changes one can quickly teach most libraries to seamlessly interact with all the file systems supported by fsspec.

This shim layer would mandate strict adherence to standard library semantics for local file operations, likely by directly forwarding all local paths into the standard library and forwarding non-local paths through fsspec-based implementations. The explicit goal would be to enable a majority of basic use cases, deferring to fsspec interfaces for more robust integration and/or specialized use cases. This would turn fsspec into a massively useful layer for updating existing systems to cloud-compatible storage, as updating a library to support s3 and gcsfs would be as simple as:

try:
    from fsspec.stdlib import open
    import fsspec.stdlib.os.path as os.path
    import fsspec.stdlib.shutil as shutil
except ImportError:
    import os.path
    import shutil

[martindurant]

Seems a reasonable thing to try - I don't know how much demand there is for this. Of course, I planned the API of the library to try to be a reasonable set of methods with names and functionality inspired from a number of sources, the stdlib being just one.

[asford]

Agreed on the existing demand, there's n=3 folks I work with but this may not be a widespread solution. Would you be open to PR here for this idea?

[asford]

I've prototyped this a bit more internally think the best approach would be a fsspec-backed implementation of the pathlib.Path interface. There's already some prior art implementing this interface for http url paths, which could be generalized comparatively easily for fsspec paths.

Would you be open to having that land in fsspec?

[martindurant]

Sure, I'm not opposed so long as it doesn't affect non-Path users. I don't think there are other people asking for it, but I could easily be wrong.

[asford]

Yes, this would be a layer sitting above the current fsspec implementation and wouldn't impact the existing APIs. I don't think there's a major outstanding call for this; I'm primarily looking for a logical spot for features I've already built over fsspec and think this would be a logical (and discoverable) home.

[ghost]

+1 - I've been using pathlib pretty heavily and just found / started using fsspec, so I'd find a pathlib compatible API shim quite useful.

[martindurant]

Perhaps between the two of you you could work out something?

[ghost]

Here's a (mostly) pathlib compatible wrapper that I'm using. I take it there's not a ton of interest, so I'll just throw it out there as a gist in case it is of use to anyone.

[martindurant]

That is interesting. If you would like to post it as a PR, it may get more attention - if you feel it is mostly ready for the public, of course.

[quantology]

fwiw, I'm using this gist to shim pytest-datadir to work with s3. I'd certainly +1 this being included in fsspec or being its own module.

[martindurant]

Yes, please, someone wrap this as a PR.