Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Proof of concept fsspec integration #109

Closed
wants to merge 3 commits into from
Closed

[WIP] Proof of concept fsspec integration #109

wants to merge 3 commits into from

Conversation

jayqi
Copy link
Member

@jayqi jayqi commented Nov 16, 2020

Implementation of FsspecClient and FsspecPath that work with an fsspec concrete filesystem implementation.

Follow up on #96


  • New abstract FsspecClient, FsspecPath classes. Each registered implementation in fsspec.registry has subclasses autogenerated, e.g., S3Path, GCSPath, HTTPPath.
  • FsspecPath will automatically dispatch to correct subclass, the same way CloudPath does. This dispatching is isolated to FsspecPath; CloudPath still only dispatches to original implementations.
    • Running environment variable FSSPEC_MODE=1 will replace CloudPath's dispatching so that it will dispatch to fsspec subclasses instead of the normal ones.
  • New extras.
    • [fsspec] will install fsspec.
    • Each extras from fsspec has been copied to [fsspec-<name>], e.g.fsspec-s3fs.
    • [fsspec-all] will install all fsspec extras.
    • original [all] does not include any fsspec extras

@jayqi jayqi mentioned this pull request Nov 16, 2020
@jayqi jayqi requested a review from pjbull November 20, 2020 06:17
@github-actions
Copy link
Contributor

@codecov
Copy link

codecov bot commented Nov 20, 2020

Codecov Report

Merging #109 (ba3d9f3) into master (8b230c3) will decrease coverage by 12.3%.
The diff coverage is 15.7%.

@@            Coverage Diff            @@
##           master    #109      +/-   ##
=========================================
- Coverage    91.4%   79.1%   -12.4%     
=========================================
  Files           8      12       +4     
  Lines         680     792     +112     
=========================================
+ Hits          622     627       +5     
- Misses         58     165     +107     
Impacted Files Coverage Δ
cloudpathlib/fsspec/__init__.py 0.0% <0.0%> (ø)
cloudpathlib/fsspec/fsspecclient.py 0.0% <0.0%> (ø)
cloudpathlib/fsspec/fsspecpath.py 0.0% <0.0%> (ø)
cloudpathlib/fsspec/implementations.py 0.0% <0.0%> (ø)
cloudpathlib/cloudpath.py 89.8% <100.0%> (+0.1%) ⬆️

@pjbull
Copy link
Member

pjbull commented Jan 22, 2023

See discussion in #96

@pjbull pjbull closed this Jan 22, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants