(feat): `xarray` with experimental `backed` reading #1247

ilan-gold · 2023-11-30T12:51:54Z

This PR is a lighter weight version of #947 that involves using the original AnnData object as the class to hold obs and var xr.Dataset.

Closes Dask and Zarr not loading obsp and obsm from remote s3 #951 and closes lazy dataframes in .obs and .var with backed="r" mode #981
Tests added
Release note added (or unnecessary)

codecov · 2023-12-07T16:24:55Z

Codecov Report

Attention: Patch coverage is 88.81356% with 33 lines in your changes missing coverage. Please review.

Project coverage is 84.57%. Comparing base (34e9783) to head (bf710d0).

Files with missing lines	Patch %	Lines
src/anndata/experimental/backed/_lazy_arrays.py	86.15%	9 Missing ⚠️
src/anndata/_core/storage.py	37.50%	5 Missing ⚠️
src/anndata/tests/helpers.py	75.00%	5 Missing ⚠️
src/anndata/_io/specs/lazy_methods.py	91.30%	4 Missing ⚠️
src/anndata/experimental/backed/_compat.py	86.20%	4 Missing ⚠️
src/anndata/experimental/backed/_io.py	89.47%	4 Missing ⚠️
src/anndata/_core/aligned_df.py	80.00%	1 Missing ⚠️
src/anndata/_core/index.py	80.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1247      +/-   ##
==========================================
- Coverage   86.87%   84.57%   -2.31%     
==========================================
  Files          39       44       +5     
  Lines        6036     6308     +272     
==========================================
+ Hits         5244     5335      +91     
- Misses        792      973     +181

Flag	Coverage Δ
	`82.81% <84.06%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
src/anndata/_core/anndata.py	`83.77% <100.00%> (+0.04%)`	⬆️
src/anndata/_core/merge.py	`83.91% <ø> (-11.08%)`	⬇️
src/anndata/_core/views.py	`85.71% <100.00%> (-5.40%)`	⬇️
src/anndata/_io/specs/__init__.py	`100.00% <ø> (ø)`
src/anndata/_io/specs/registry.py	`95.53% <100.00%> (-0.50%)`	⬇️
src/anndata/_io/zarr.py	`83.75% <100.00%> (+0.20%)`	⬆️
src/anndata/_types.py	`85.29% <100.00%> (ø)`
src/anndata/experimental/__init__.py	`100.00% <100.00%> (ø)`
src/anndata/experimental/backed/__init__.py	`100.00% <100.00%> (ø)`
src/anndata/experimental/backed/_xarray.py	`100.00% <100.00%> (ø)`
... and 8 more

... and 4 files with indirect coverage changes

Co-authored-by: Philipp A. <[email protected]>

for more information, see https://pre-commit.ci

…set_kwargs`

…g/read_dask_elem

Co-authored-by: Philipp A. <[email protected]>

…erator_from_elems`

ivirshup · 2024-09-17T23:45:32Z

/azp run

azure-pipelines · 2024-09-17T23:45:38Z

Pull request contains merge conflicts.

src/anndata/experimental/backed/_io.py

ivirshup · 2024-08-14T22:52:17Z

src/anndata/experimental/backed/_io.py

+        f = h5py.File(store, mode="r")
+
+    def callback(func, elem_name: str, elem, iospec):
+        if iospec.encoding_type == "anndata" or elem_name.endswith("/"):


What's the elem_name.endswith("/") case?

Presumably root store, we use this in all the other read methods...

ivirshup · 2024-09-17T23:48:37Z

src/anndata/experimental/backed/_compat.py

+try:
+    import xarray as xr
+except ImportError:
+    xr = None
+
+
+try:
+    from xarray.backends.zarr import ZarrArrayWrapper
+except ImportError:
+
+    class ZarrArrayWrapper:
+        def __repr__(self) -> str:
+            return "mock ZarrArrayWrapper"


This code means that you get a really confusing error if you run read_backed without xarray being installed

Yes but unfortunately I think this compat layer is needed so that core is compatible. Maybe we can add a check within read_backed?

ebezzi · 2024-09-18T17:15:56Z

Issue: if you run adata.obs["cell_type"].to_pandas() (on a dataset from cellxgene), the output should be a categorical, but it is actually an object:

obs_names
0                                   macrophage
1                                 stromal cell
2       CD8-positive, alpha-beta memory T cell
3       CD8-positive, alpha-beta memory T cell
4        CD141-positive myeloid dendritic cell
                         ...                  
9419                              stromal cell
9420                              stromal cell
9421                              stromal cell
9422    CD8-positive, alpha-beta memory T cell
9423                              stromal cell
Length: 9424, dtype: object

ivirshup · 2024-09-18T18:55:21Z

src/anndata/_core/merge.py

@@ -757,6 +757,7 @@ def np_bool_to_pd_bool_array(df: pd.DataFrame):
    return df


+# TODO: concat for xarray


This would be nice to have with this PR. What are your thoughts on this @ilan-gold?

Yea......I agree......

Ok, sorry, I remember now. I looked into this...it's a lot of work...but maybe I'll try my hand at it....sometimes you get fatigue just from how daunting something seems

i can try to help if you open a PR showing where you need input

Ah @dcherian I am actually almost 100% sure this is already implemented in xarray, but we need to get the functionality here! I think it's all good on your end but I will let you know if we need anything!

ilan-gold · 2024-09-19T10:30:47Z

@ivirshup @ebezzi filed pydata/xarray#9519 but it should not block if it does not come out before this does IMO.

And now: pydata/xarray#9520

ilan-gold mentioned this pull request Nov 30, 2023

Dask and Zarr not loading obsp and obsm from remote s3 #951

Open

ilan-gold mentioned this pull request Jan 31, 2024

lazy dataframes in .obs and .var with backed="r" mode #981

Open

ilan-gold and others added 18 commits June 3, 2024 14:11

(chore): sort indices to prevent warning

3672c18

(fix): remove utility function make_dask_array

33c3599

(chore): read_sparse_as_dask_h5 -> read_sparse_as_dask

157e710

(feat): make params of h5_chunks and stride

375000d

(chore): add distributed test

241904a

(fix): TypeVar bind

42d0d22

(chore): release note

0bba2c0

(chore): 0.10.8 -> 0.11.0

0d0b43a

Merge branch 'main' into ig/read_dask_elem

762d4c6

(fix): ruff for default pytest.fixture scope

c935fe0

Apply suggestions from code review

23e0ea2

Co-authored-by: Philipp A. <[email protected]>

(fix): Any to DaskArray

5b96c77

(fix): type make_index + fix undeclared

0907a4e

[pre-commit.ci] auto fixes from pre-commit.com hooks

20ced16

for more information, see https://pre-commit.ci

Merge branch 'main' into ig/read_dask_elem

36ae8f2

fix rest

bb6607e

(fix): use chunks kwarg

419691b

Merge branch 'main' into ig/read_dask_elem

a23df34

ilan-gold added this to the 0.11.0 milestone Jul 2, 2024

ilan-gold self-assigned this Jul 2, 2024

ilan-gold added 7 commits July 2, 2024 15:58

(feat): expose chunks as an option to read_elem_as_dask via `data…

fd2376a

…set_kwargs`

Merge branch 'ig/read_dask_elem' of github.com:scverse/anndata into i…

ae723d0

…g/read_dask_elem

(fix): test_read_dispatched_null_case test

42b1093

(fix): disallowed spread syntax?

78de057

(refactor): reuse compute_chunk_layout_for_axis_shape functionality

717b997

(fix): upgrade xarray for new version

14a2440

Merge branch 'main' into ig/xarray_compat

2aa6420

flying-sheep and others added 13 commits August 30, 2024 13:51

Merge branch 'main' into ig/xarray_compat

4587fec

(fix): should -> shall

97eace5

Apply suggestions from code review

58654a8

Co-authored-by: Philipp A. <[email protected]>

(fix): _gen_xarray_dict_iterator_from_elems -> `_gen_xarray_dict_it…

67af64f

…erator_from_elems`

(feat): indexing with DataArray

bfc2e73

Merge branch 'main' into ig/xarray_compat

b801d35

(fix): check h5 store

a1d0b89

Merge branch 'main' into ig/xarray_compat

160b522

(fix):check DataArray closer

5f80b61

(fix): clean up api.md from merge

8ce6409

(fix): remove read_elem_as_dask docs reference

f9ef9f0

(chore): add notebooks/read_backed_experimental

5e69a50

(chore): update notebooks

1540d27

ilan-gold force-pushed the ig/xarray_compat branch from 4947295 to 1540d27 Compare September 4, 2024 14:34

ivirshup reviewed Sep 17, 2024

View reviewed changes

ilan-gold added 9 commits September 18, 2024 16:12

(refactor): set pytestmark at the top

371fc2b

(chore): clarify comment

5cb2d8d

(refactor): add assert_access_count method for AccessTrackingStore

b86ee6b

(refactor): read_backed->read_lazy

227a3c6

(fix): actually read_backed -> read_lazy

3debf9b

(chore): time to require aiohttp fsspec and zarr and requests

4b48988

Merge branch 'main' into ig/xarray_compat

28c95a7

(chore): update notebook

cec633a

Merge branch 'main' into ig/xarray_compat

97aff04

ivirshup reviewed Sep 18, 2024

View reviewed changes

(fix): actually only read index once

bf710d0

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

(feat): `xarray` with experimental `backed` reading #1247

(feat): `xarray` with experimental `backed` reading #1247

ilan-gold commented Nov 30, 2023 •

edited by ivirshup

Loading

codecov bot commented Dec 7, 2023 •

edited

Loading

ivirshup commented Sep 17, 2024

azure-pipelines bot commented Sep 17, 2024

ivirshup Aug 14, 2024

ilan-gold Sep 18, 2024

ivirshup Sep 17, 2024

ilan-gold Sep 18, 2024

ebezzi commented Sep 18, 2024

ivirshup Sep 18, 2024

ilan-gold Sep 19, 2024

ilan-gold Sep 19, 2024

dcherian Sep 19, 2024

ilan-gold Sep 19, 2024

ilan-gold commented Sep 19, 2024 •

edited

Loading

		@@ -757,6 +757,7 @@ def np_bool_to_pd_bool_array(df: pd.DataFrame):
		return df


		# TODO: concat for xarray

(feat): xarray with experimental backed reading #1247

Are you sure you want to change the base?

(feat): xarray with experimental backed reading #1247

Conversation

ilan-gold commented Nov 30, 2023 • edited by ivirshup Loading

codecov bot commented Dec 7, 2023 • edited Loading

Codecov Report

ivirshup commented Sep 17, 2024

azure-pipelines bot commented Sep 17, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ebezzi commented Sep 18, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ilan-gold commented Sep 19, 2024 • edited Loading

(feat): `xarray` with experimental `backed` reading #1247

(feat): `xarray` with experimental `backed` reading #1247

ilan-gold commented Nov 30, 2023 •

edited by ivirshup

Loading

codecov bot commented Dec 7, 2023 •

edited

Loading

ilan-gold commented Sep 19, 2024 •

edited

Loading