FileNotFoundError in yokogawa-to-zarr task #280
Comments
|
Versions: |
|
Quick comments:
|
|
It seems that going back with zarr version made the difference. This set of versions works: dask 2022.7.1, https://docs.dask.org/en/stable/changelog.html Main suspect (for the moment): https://zarr.readthedocs.io/en/stable/release.html#enhancements |
|
Other info about zarr versions:
EDIT:
|
|
Possible candidate: canvas_zarr = zarr.create(
shape=(len(wavelength_ids), max_z, max_y, max_x),
chunks=chunksize,
dtype=sample.dtype,
store=da.core.get_mapper(zarrurl + "/0"),
overwrite=False,
dimension_separator="/",
)Refs:
Even more in detail, I'm now looking at this diff: |
|
Wow, thanks for digging into this @tcompa ! Very good if we can solve then underlying TaskError as well and pity that it's a dependency thing once again... 🙈 |
|
Also, on the quite confusing side: It still appears to have created a correct OME-Zarr file for me, even though it may be triggering this task error. Have you checked the Zarr files you're getting with the error? |
|
More debugging: This may be the important new zarr codeblock that changed: With zarr 2.13.6 (the recent one, that doesn't work), we end up in the FSS branch: FSStore._fsspec_installed(): True
isinstance(store, fsspec.FSMap): Trueand then store: <FSStore({
'.zarray': (
b'{\n'
b' "chunks": [\n'
b' 1,\n'
b' 1,\n'
b' 2160,\n'
b' 2560\n'
b' ],\n'
b' "compressor": {\n'
b' "blocksize": 0,\n'
b' "clevel": 5,\n'
b' "cname": "lz4",\n'
b' "id": "blosc",\n'
b' "shuffle": 1\n'
b' },\n'
b' "dimension_separator": "/",\n'
b' "dtype": "<u2",\n'
b' "fill_value": 0,\n'
b' "filters": null,\n'
b' "order": "C",\n'
b' "shape": [\n'
b' 3,\n'
b' 10,\n'
b' 4320,\n'
b' 5120\n'
b' ],\n'
b' "zarr_format": 2\n'
b'}'
),
})> (FSStore) len=1And then something fails.. When using previous versions, I think we ended up in the |
|
For the record, fsspec.get_mapper() (which we are currently using) creates a I'm not sure of why we would need that store, and cannot use a more standard |
|
Additional info: the CI correctly fails, as soon as we switch to zarr 2.3.16 - see for instance And see also: https://github.com/fractal-analytics-platform/fractal-tasks-core/actions/runs/3939719186/jobs/6739903447 This means we also have a way to test new implementations, for instance based on |
|
Finally a minimal non-working example: import os
import shutil
import dask.array as da
x = da.ones((2, 4), chunks=(1, 2))
print(x)
zarrurl = "/tmp/plate_ones.zarr/"
if os.path.isdir(zarrurl):
shutil.rmtree(zarrurl)
x.to_zarr(zarrurl, dimension_separator="/")Running this script with zarr 2.13.6 leads to the usual traceback Traceback (most recent call last):
File "/home/tommaso/Fractal/fractal-tasks-core/tests/data/mwe.py", line 12, in <module>
x.to_zarr(zarrurl, dimension_separator="/")
File "/home/tommaso/miniconda3/envs/test/lib/python3.9/site-packages/dask/array/core.py", line 2828, in to_zarr
return to_zarr(self, *args, **kwargs)
File "/home/tommaso/miniconda3/envs/test/lib/python3.9/site-packages/dask/array/core.py", line 3591, in to_zarr
return arr.store(z, lock=False, compute=compute, return_stored=return_stored)
File "/home/tommaso/miniconda3/envs/test/lib/python3.9/site-packages/dask/array/core.py", line 1752, in store
r = store([self], [target], **kwargs)
File "/home/tommaso/miniconda3/envs/test/lib/python3.9/site-packages/dask/array/core.py", line 1226, in store
compute_as_if_collection(Array, store_dsk, map_keys, **kwargs)
File "/home/tommaso/miniconda3/envs/test/lib/python3.9/site-packages/dask/base.py", line 347, in compute_as_if_collection
return schedule(dsk2, keys, **kwargs)
File "/home/tommaso/miniconda3/envs/test/lib/python3.9/site-packages/dask/threaded.py", line 89, in get
results = get_async(
File "/home/tommaso/miniconda3/envs/test/lib/python3.9/site-packages/dask/local.py", line 511, in get_async
raise_exception(exc, tb)
File "/home/tommaso/miniconda3/envs/test/lib/python3.9/site-packages/dask/local.py", line 319, in reraise
raise exc
File "/home/tommaso/miniconda3/envs/test/lib/python3.9/site-packages/dask/local.py", line 224, in execute_task
result = _execute_task(task, data)
File "/home/tommaso/miniconda3/envs/test/lib/python3.9/site-packages/dask/core.py", line 119, in _execute_task
return func(*(_execute_task(a, cache) for a in args))
File "/home/tommaso/miniconda3/envs/test/lib/python3.9/site-packages/dask/array/core.py", line 4260, in store_chunk
return load_store_chunk(x, out, index, lock, return_stored, False)
File "/home/tommaso/miniconda3/envs/test/lib/python3.9/site-packages/dask/array/core.py", line 4242, in load_store_chunk
out[index] = x
File "/home/tommaso/miniconda3/envs/test/lib/python3.9/site-packages/zarr/core.py", line 1373, in __setitem__
self.set_basic_selection(pure_selection, value, fields=fields)
File "/home/tommaso/miniconda3/envs/test/lib/python3.9/site-packages/zarr/core.py", line 1468, in set_basic_selection
return self._set_basic_selection_nd(selection, value, fields=fields)
File "/home/tommaso/miniconda3/envs/test/lib/python3.9/site-packages/zarr/core.py", line 1772, in _set_basic_selection_nd
self._set_selection(indexer, value, fields=fields)
File "/home/tommaso/miniconda3/envs/test/lib/python3.9/site-packages/zarr/core.py", line 1844, in _set_selection
self._chunk_setitems(lchunk_coords, lchunk_selection, chunk_values,
File "/home/tommaso/miniconda3/envs/test/lib/python3.9/site-packages/zarr/core.py", line 2046, in _chunk_setitems
self.chunk_store.setitems(to_store)
File "/home/tommaso/miniconda3/envs/test/lib/python3.9/site-packages/zarr/storage.py", line 1394, in setitems
self.map.setitems(values)
File "/home/tommaso/miniconda3/envs/test/lib/python3.9/site-packages/fsspec/mapping.py", line 116, in setitems
self.fs.pipe(values)
File "/home/tommaso/miniconda3/envs/test/lib/python3.9/site-packages/fsspec/spec.py", line 757, in pipe
self.pipe_file(self._strip_protocol(k), v, **kwargs)
File "/home/tommaso/miniconda3/envs/test/lib/python3.9/site-packages/fsspec/spec.py", line 736, in pipe_file
with self.open(path, "wb", **kwargs) as f:
File "/home/tommaso/miniconda3/envs/test/lib/python3.9/site-packages/fsspec/spec.py", line 1106, in open
f = self._open(
File "/home/tommaso/miniconda3/envs/test/lib/python3.9/site-packages/fsspec/implementations/local.py", line 175, in _open
return LocalFileOpener(path, mode, fs=self, **kwargs)
File "/home/tommaso/miniconda3/envs/test/lib/python3.9/site-packages/fsspec/implementations/local.py", line 273, in __init__
self._open()
File "/home/tommaso/miniconda3/envs/test/lib/python3.9/site-packages/fsspec/implementations/local.py", line 278, in _open
self.f = open(self.path, mode=self.mode)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/plate_ones.zarr/0/0'There at least three ways of getting the script to work:
|
|
Let's see if we receive some useful advice on dask/dask#9841. |
|
Temporary fix: let's update dask to 2023.1, which works fine with zarr 2.13.6 (see issue on dask repo). |
…underror-in-yokogawa-to-zarr-task Update zarr and dask (ref #280)
Ref:
TL;DR
Dask tries to access
20200812-CardiomyocyteDifferentiation14-Cycle1.zarr/B/03/0//0/0/0/0/0', which is obviously wrong (two slashes and too many hierarchy levels - possibly related?1)More info:
02_cardio_smallexample of the fractal-demos repo.run_example.shis the same as on github, apart from removing all tasks that take place after"Convert Yokogawa to OME-Zarr".The text was updated successfully, but these errors were encountered: