Create fsstore from filesystem #911

rabernat · 2021-12-14T15:08:18Z

This PR adds a new argument to FSStore which allows one to pass an existing fsspec filesystem object from which to generate the FSStore. The need to initialize an FSStore from an existing filesystem came up in pangeo-forge/pangeo-forge-recipes#254. It seems like this is something people may want to do more generally.

Also fixes #993 by simply removing the check for list-ability when initializing an FSStore.

TODO:

Add unit tests and/or doctests in docstrings
Add docstrings and API docs for any new/modified user-facing classes and functions
New/modified features documented in docs/tutorial.rst
Changes documented in docs/release.rst
GitHub Actions have all passed
Test coverage is 100% (Codecov passes)

zarr/storage.py

zarr/tests/test_storage.py

codecov · 2021-12-14T15:19:32Z

Codecov Report

Merging #911 (c1c9255) into master (d1f590d) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master     #911   +/-   ##
=======================================
  Coverage   99.94%   99.94%           
=======================================
  Files          34       34           
  Lines       13719    13785   +66     
=======================================
+ Hits        13712    13778   +66     
  Misses          7        7

Impacted Files	Coverage Δ
zarr/storage.py	`100.00% <100.00%> (ø)`
zarr/tests/test_convenience.py	`100.00% <100.00%> (ø)`
zarr/tests/test_core.py	`100.00% <100.00%> (ø)`
zarr/tests/test_storage.py	`100.00% <100.00%> (ø)`

rabernat · 2022-01-31T18:45:16Z

I would like to revive and move forward this PR. Martin, you gave a few helpful comments and ideas above. Would you mind turning these into a more formal review in terms of what you think is needed vs interesting possibility.

What we have here is a tightly scoped and tested PR to unblock a rather serious upstream problem. If there is no major technical debt incurred by this PR, could we consider merging as is (modulo doc updates from checklist) rather than expanding scope? If not, please let us know what else would be required.

Also pinging @grlee77 and @joshmoore for potential interaction with ongoing refactor of storage classes (#789).

grlee77 · 2022-01-31T19:08:23Z

Also pinging @grlee77 and @joshmoore for potential interaction with ongoing refactor of storage classes (#789).

Thanks @rabernat, I commented above. If the __init__ signature does not change then this should be pretty easy to merge into the work there.

joshmoore · 2022-02-03T20:58:46Z

Thanks, @grlee77. From my side, that was the only concern in terms of impact on existing work, @rabernat. @martindurant's point about the general API "burden" seems to be outstanding. Is it that a concrete alternative that doesn't break the API needs proposing from your side, @rabernat?

A side note that testing is currently blocked due to GHA weirdness. @jakirkham and I will try to get that sorted ASAP.

joshmoore · 2022-03-07T22:10:34Z

@rabernat: would you prefer someone else to propose an alternative?

rabernat · 2022-04-04T16:26:04Z

zarr/storage.py

-        if self.fs.exists(self.path) and not self.fs.isdir(self.path):
-            raise FSPathExistNotDir(url)


To fix #993 I have simply removed this check...

rabernat · 2022-04-04T16:28:30Z

zarr/tests/test_storage.py

-        filepath = os.path.join(path, self.root + "foo")
-        with pytest.raises(ValueError):
-            self.create_store(path=filepath, mode='r')


...and its corresponding test.

This test did not do anything other than test the error above. self.create_store does not actually attempt to write any data or list the store.

With the check gone, users will now see an error from fsspec if they try to do an operation that requires listing on an unlistable FSSpec store, as they would already on a generic MutableMapping-based store.

pep8speaks · 2022-04-05T13:13:20Z

Hello @rabernat! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2022-05-04 11:10:02 UTC

martindurant

very little to comment

zarr/storage.py

zarr/tests/test_convenience.py

rabernat · 2022-04-07T14:48:19Z

docs/release.rst

+2.11.3
+------


Although 2.11.3 has been released, I noted that the docs had not been updated, so I added it here.

It seems we have skipped release 2.11.2?

Those releases are coming from the 2_11 branch. (See the related #898 (comment))

I assume you intend this for release as a 2.11.x?

Sorry I have clearly not been able to follow that discussion. After reading, I now understand the situation better. Let's discuss at today's SC meeting.

Please ping this thread with any decision on which branch to merge into

@martindurant : @rabernat seemed to think it was ok to hold off on @grlee77's upcoming v3 configuration fix, i.e. this can stay on the branch it is and be released as something >2.11.x. If it looks like we want it in a quick ~=2.11.x release, I can attempt the backport.

I will do whatever the other devs recommend here (either backport to 2.11.x or put this in 2.12.0)

I am confused about how we will maintain a consistent changelog for these two branches. So I am standing by for advice.

rabernat · 2022-04-07T14:48:55Z

From my POV, this is now complete. Would appreciate any other feedback or otherwise a merge.

martindurant · 2022-04-07T15:05:35Z

@joshmoore , I see you have been tagged to review. I am happy to put this in, if you don't intend to look.

rabernat · 2022-04-07T15:09:15Z

This is my first Zarr PR in quite a while, so I just tagged the people I know to review. 😄

rabernat · 2022-04-21T16:29:35Z

I feel confused about how to move forward with this PR in terms of how to update the changelog (see #911 (comment)). Please help me 🙏. I would love to see this merged soon as I believe it is finished. It is now starting to accumulate conflicts.

joshmoore · 2022-04-21T19:19:48Z

Sorry for having left you hanging. I think either will be fine @rabernat. Either just add your item to unreleased or copy over from the 2 11 branch. Either way feel free to get this in and I will double check before the next release. (Of whatever branch)

joshmoore

I've hopefully handled the conflicts with Greg's v3 work. Once this goes green, I'll merge and update the release notes to include previous versions.

joshmoore · 2022-05-04T10:51:15Z

It looks like the semantics of the update() method have changed with the refactoring for v3:

previous noise

pytest zarr/tests/test_convenience.py -svx -k2

...snip...

>           store_to_open.update(store)  # copy original store to new unlistable store

zarr/tests/test_convenience.py:305:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <zarr.storage.FSStore object at 0x1ba010970>, other = '/var/folders/z5/txc_jj6x5l5cm81r56ck1n9c0000gn/T/tmp0gy4ji7i', kwds = {}

    def update(self, other=(), /, **kwds):
        ''' D.update([E, ]**F) -> None.  Update D from mapping/iterable E and F.
            If E present and has a .keys() method, does:     for k in E: D[k] = E[k]
            If E present and lacks .keys() method, does:     for (k, v) in E: D[k] = v
            In either case, this is followed by: for k, v in F.items(): D[k] = v
        '''
        if isinstance(other, Mapping):
            for key in other:
                self[key] = other[key]
        elif hasattr(other, "keys"):
            for key in other.keys():
                self[key] = other[key]
        else:
>           for key, value in other:
E           ValueError: not enough values to unpack (expected 2, got 1)

That doesn't appear to be related to the type hierarchy:

2.11.2: FSStore.__mro__: (<class 'zarr.storage.FSStore'>, <class 'zarr._storage.store.Store'>, <class 'zarr._storage.store.BaseStore'>, <class 'collections.abc.MutableMapping'>, <class 'collections.abc.Mapping'>, <class 'collections.abc.Collection'>, <class 'collections.abc.Sized'>, <class 'collections.abc.Iterable'>, <class 'collections.abc.Container'>, <class 'object'>)
2.12.x: FSStore.__mro__: (<class 'zarr.storage.FSStore'>, <class 'zarr._storage.store.Store'>, <class 'zarr._storage.store.BaseStore'>, <class 'collections.abc.MutableMapping'>, <class 'collections.abc.Mapping'>, <class 'collections.abc.Collection'>, <class 'collections.abc.Sized'>, <class 'collections.abc.Iterable'>, <class 'collections.abc.Container'>, <class 'object'>)

Still digging in case anyone has an idea.

Edit: the problem was just that store had become a str. 🤦🏽 Fix pushed.

rabernat · 2022-05-04T12:58:40Z

Thanks for finishing this up Josh! I really appreciate it!

rabernat requested a review from martindurant December 14, 2021 15:08

rabernat mentioned this pull request Dec 14, 2021

finalize_target is slow on large recipes with remote filesystem #987 pangeo-forge/pangeo-forge-recipes#254

Open

martindurant reviewed Dec 14, 2021

View reviewed changes

zarr/storage.py Show resolved Hide resolved

zarr/storage.py Show resolved Hide resolved

zarr/tests/test_storage.py Show resolved Hide resolved

rabernat mentioned this pull request Jan 31, 2022

Avoid costly set(group) operation in finalize_target. pangeo-forge/pangeo-forge-recipes#265

Merged

rabernat requested review from d-v-b and joshmoore and removed request for d-v-b January 31, 2022 18:45

rabernat mentioned this pull request Mar 28, 2022

Permission error with exists on a bucket (OSError storage.buckets.get scope needed) fsspec/gcsfs#462

Closed

rabernat mentioned this pull request Apr 4, 2022

Add all FSMap constructor arguments as class attributes fsspec/filesystem_spec#939

Merged

rabernat commented Apr 4, 2022

View reviewed changes

rabernat requested a review from martindurant April 5, 2022 15:27

martindurant reviewed Apr 5, 2022

View reviewed changes

zarr/storage.py Show resolved Hide resolved

zarr/tests/test_convenience.py Show resolved Hide resolved

zarr/tests/test_convenience.py Show resolved Hide resolved

rabernat force-pushed the create-fsstore-from-filesystem branch from e76a376 to cfadc1d Compare April 7, 2022 14:46

rabernat commented Apr 7, 2022

View reviewed changes

rabernat added 7 commits April 12, 2022 15:53

refactor FSStore class to allow fs argument

2865d53

add tests

05cf7a3

fixes zarr-developers#993

a5acfbf

add fsspec mapper kwargs to FSMap constructor

6756862

avoid passing missing_exceptions if possible

1b0c084

fix line length

585e16c

add tests for array creation with existing fs

5b073b4

rabernat added 5 commits April 12, 2022 15:53

add test for consolidated reading of unlistable store

3ec6855

flake8

c2ed221

rename functions and skip coverage for workaround we expect to remove

d1d8e39

update release notes and tutorial

0599d4e

fix sphinx ref typo

36c05ed

rabernat force-pushed the create-fsstore-from-filesystem branch from 3bc0299 to 36c05ed Compare April 12, 2022 19:56

Merge branch 'master' into create-fsstore-from-filesystem

c045d52

joshmoore approved these changes May 4, 2022

View reviewed changes

joshmoore added 2 commits May 4, 2022 13:09

Fix use of store.update()

0cd11c7

Flake8 corrections

c1c9255

joshmoore merged commit ab6b355 into zarr-developers:master May 4, 2022

rabernat mentioned this pull request Sep 1, 2022

OSError: {'We encountered an internal error. Please try again.'} during finalize_target. pangeo-forge/pangeo-forge-recipes#406

Closed

rabernat mentioned this pull request Oct 13, 2022

speed to write files to s3 storage pangeo-data/pangeo-eosc#39

Open

rabernat mentioned this pull request Dec 15, 2022

Read performance regression via Store backed by FSMap (fsspec, GCS) #1296

Closed

tcompa mentioned this pull request Jan 17, 2023

FileNotFoundError in yokogawa-to-zarr task fractal-analytics-platform/fractal-tasks-core#280

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create fsstore from filesystem #911

Create fsstore from filesystem #911

rabernat commented Dec 14, 2021 •

edited

Loading

codecov bot commented Dec 14, 2021 •

edited

Loading

rabernat commented Jan 31, 2022 •

edited

Loading

grlee77 commented Jan 31, 2022

joshmoore commented Feb 3, 2022

joshmoore commented Mar 7, 2022

rabernat Apr 4, 2022

rabernat Apr 4, 2022

pep8speaks commented Apr 5, 2022 •

edited

Loading

martindurant left a comment

rabernat Apr 7, 2022

joshmoore Apr 7, 2022

rabernat Apr 7, 2022

martindurant Apr 7, 2022

joshmoore Apr 7, 2022

rabernat Apr 12, 2022

rabernat commented Apr 7, 2022

martindurant commented Apr 7, 2022

rabernat commented Apr 7, 2022

rabernat commented Apr 21, 2022 •

edited

Loading

joshmoore commented Apr 21, 2022

joshmoore left a comment

joshmoore commented May 4, 2022 •

edited

Loading

rabernat commented May 4, 2022

		if self.fs.exists(self.path) and not self.fs.isdir(self.path):
		raise FSPathExistNotDir(url)

		2.11.3
		------

Create fsstore from filesystem #911

Create fsstore from filesystem #911

Conversation

rabernat commented Dec 14, 2021 • edited Loading

codecov bot commented Dec 14, 2021 • edited Loading

Codecov Report

rabernat commented Jan 31, 2022 • edited Loading

grlee77 commented Jan 31, 2022

joshmoore commented Feb 3, 2022

joshmoore commented Mar 7, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pep8speaks commented Apr 5, 2022 • edited Loading

Comment last updated at 2022-05-04 11:10:02 UTC

martindurant left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rabernat commented Apr 7, 2022

martindurant commented Apr 7, 2022

rabernat commented Apr 7, 2022

rabernat commented Apr 21, 2022 • edited Loading

joshmoore commented Apr 21, 2022

joshmoore left a comment

Choose a reason for hiding this comment

joshmoore commented May 4, 2022 • edited Loading

rabernat commented May 4, 2022

rabernat commented Dec 14, 2021 •

edited

Loading

codecov bot commented Dec 14, 2021 •

edited

Loading

rabernat commented Jan 31, 2022 •

edited

Loading

pep8speaks commented Apr 5, 2022 •

edited

Loading

rabernat commented Apr 21, 2022 •

edited

Loading

joshmoore commented May 4, 2022 •

edited

Loading