ReadOnlyError when opening a v3 store

[jhamman]

Zarr version

v2.14.2

Numcodecs version

v0.11.0

Python Version

3.10.8

Operating System

Mac (M2)

Installation

pip

Description

When opening a v3 store, it is possible to trigger zarr.errors.ReadOnlyError: object is read-only when mode='r'.

Steps to reproduce

The reproducer here is a bit tough because it requires using Fsspec+Kerchunk+ZarrV3 but I'll show a traceback and point to a traceback and a where the offending code is in Zarr:

Traceback:

/Users/jhamman/miniforge3/envs/zarr-testlib/python3.10/site-packages/xarray/backends/api.py:526: in open_dataset
    backend_ds = backend.open_dataset(
/Users/jhamman/miniforge3/envs/zarr-testlib/python3.10/site-packages/xarray/backends/zarr.py:891: in open_dataset
    store = ZarrStore.open_group(
/Users/jhamman/miniforge3/envs/zarr-testlib/python3.10/site-packages/xarray/backends/zarr.py:430: in open_group
    zarr_group = zarr.open_group(store, **open_kwargs)
/Users/jhamman/miniforge3/envs/zarr-testlib/python3.10/site-packages/zarr/hierarchy.py:1416: in open_group
    store = _normalize_store_arg(
/Users/jhamman/miniforge3/envs/zarr-testlib/python3.10/site-packages/zarr/hierarchy.py:1288: in _normalize_store_arg
    return normalize_store_arg(store,
/Users/jhamman/miniforge3/envs/zarr-testlib/python3.10/site-packages/zarr/storage.py:181: in normalize_store_arg
    return normalize_store(store, storage_options, mode)
/Users/jhamman/miniforge3/envs/zarr-testlib/python3.10/site-packages/zarr/_storage/v3.py:622: in _normalize_store_arg_v3
    store['zarr.json'] = store._metadata_class.encode_hierarchy_metadata(None)
/Users/jhamman/miniforge3/envs/zarr-testlib/python3.10/site-packages/zarr/_storage/v3.py:115: in __setitem__
    super().__setitem__(key, value)

This happens because Zarr is trying to add a key to a store that is opened with mode='w':

zarr-python/zarr/_storage/v3.py

Lines 621 to 622 in 4b0705c

	# add default zarr.json metadata
	store['zarr.json'] = store._metadata_class.encode_hierarchy_metadata(None)

Additional output

A localized reproducer can be triggered like this:

out = {'version': 1, 'refs': {'zarr.json': '{"extensions":[],"metadata_encoding":"https:\\/\\/purl.org\\/zarr\\/spec\\/protocol\\/core\\/3.0","metadata_key_suffix":".json","zarr_format":"https:\\/\\/purl.org\\/zarr\\/spec\\/protocol\\/core\\/3.0"}', 'meta/root.group.json': '{"attributes":{"_zroot":"<zarr.hierarchy.Group \'\\/\'>","attr0":"3"}}', 'meta/root/data.array.json': '{"attributes":{"_ARRAY_DIMENSIONS":["x","y"]},"chunk_grid":{"chunk_shape":[10,10],"separator":"\\/","type":"regular"},"chunk_memory_layout":"C","data_type":">f8","extensions":[],"fill_value":null,"shape":[10,10]}', 'data/root/data/c0/0': ['memory://data.nc3', 152, 800]}}
_normalize_store_arg_v3("reference://", storage_options={"fo": out, "remote_protocol": "memory"})

The full reproducer is a test failure in Kerchunk:

• Zarr v3 support fsspec/kerchunk#292
• https://github.com/fsspec/kerchunk/actions/runs/4070352216/jobs/7184796481#step:5:763

Proposal

This could be a fairly simple fix to _normalize_store_arg_v3 where the 'zarr.json' key is only added when the store is writable:

if mode != 'r':
    store['zarr.json'] = store._metadata_class.encode_hierarchy_metadata(None) 

cc @martindurant

[rabernat]

There have been a ton of these "read only" errors recently.

• regression using fsspec.mapping.FSMap object to initialize a store #1352
• Creating an Zarr.group using fsspec.FSMap fails #1353
• Dask to_zarr now complains about read-only after #1304 merged #1306
• zarr.core.Array.chunk_store changes type after zarr=2.23.3 (after PR 1304) #1362

I thought we fixed it in 2.14.2. But maybe not fully.