ImplicitToExplicitIndexingAdapter being returned with dask unstable version

[djhoese]

What happened:

I have a couple CI environments that use unstable versions of xarray and dask among other libraries. A couple tests have been failing in different but similar ways. I was able to get one of them down to the below example. This happens with the dask master branch on either the xarray latest release or xarray master. Here are the commits I was using when doing unstable both.

Dask: ad01acc14088d03127dfec4f881cce959f3ac4d9
Xarray: eb7e112d45a9edebd8e5fb4f873e3e6adb18824a

I don't see this with dask's latest release and this is likely caused by a change in dask, but after seeing PRs like #4884 I'm wondering if this is something similar and requires a change in xarray.

The bottom line is that doing various things like my_data_arr.data.compute() or dask.compute(my_data_arr.data) produces:

ImplicitToExplicitIndexingAdapter(array=CopyOnWriteArray(array=LazilyOuterIndexedArray(array=<xarray.backends.rasterio_.RasterioArrayWrapper object at 0x7f257b382150>, key=BasicIndexer((slice(0, 2, 1), slice(0, 100, 1), slice(0, 100, 1))))))

What you expected to happen:

I would expect to get a numpy array back when computing the underlying dask array.

Minimal Complete Verifiable Example:

from PIL import Image
import xarray as xr
import numpy as np

# create a test image
Image.fromarray(np.zeros((5, 5, 3), dtype=np.uint8)).save('test.png')
r = xr.open_rasterio('test.png', chunks='auto')
print(r.data.compute())
# ImplicitToExplicitIndexingAdapter(array=CopyOnWriteArray(array=LazilyOuterIndexedArray(array=<xarray.backends.rasterio_.RasterioArrayWrapper object at 0x7f25e72bcc10>, key=BasicIndexer((slice(0, 2, 1), slice(0, 100, 1), slice(0, 100, 1))))))

Anything else we need to know?:

As mentioned above other weird things are happening when array wrappers seem to be involved but I haven't been able to make a small example of them.

Environment:

Output of xr.show_versions()

INSTALLED VERSIONS ------------------ commit: None python: 3.7.6 | packaged by conda-forge | (default, Jan 7 2020, 22:33:48) [GCC 7.3.0] python-bits: 64 OS: Linux OS-release: 5.8.0-7642-generic machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: en_US.UTF-8 libhdf5: 1.10.5 libnetcdf: 4.7.3

xarray: 0.16.3.dev132+geb7e112d
pandas: 1.2.2
numpy: 1.20.1
scipy: 1.6.1
netCDF4: 1.5.3
pydap: None
h5netcdf: None
h5py: 2.10.0
Nio: None
zarr: 2.6.2.dev42
cftime: 1.4.1
nc_time_axis: None
PseudoNetCDF: None
rasterio: 1.3dev
cfgrib: None
iris: None
bottleneck: 1.4.0.dev0+117.gf2bc792
dask: 2021.02.0+21.gad01acc1
distributed: 2021.02.0+7.g383ea032
matplotlib: 3.4.0rc1
cartopy: 0.17.0
seaborn: None
numbagg: None
pint: None
setuptools: 45.2.0.post20200209
pip: 20.0.2
conda: None
pytest: 5.3.5
IPython: 7.12.0
sphinx: 2.4.3

[djhoese]

Should I make an issue with dask so this gets more eyes before they make their release (I'm told later this week)?

[dcherian]

cc @crusaderky

[shoyer]

Should I make an issue with dask so this gets more eyes before they make their release (I'm told later this week)?

Yes, please!

ImplicitToExplicitIndexingAdapter is an example of an "input array" rather than a full "duck array." It's true that we don't want to eagerly convert these into NumPy arrays, since they can be lazily indexed, but on the other hand it's not an array type that users know what to deal with, so it's weird to get them back from compute().

[djhoese]

dask/dask#7263