Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support arbitrary block index functions in blockwise #350

Merged
merged 1 commit into from
Jan 18, 2024

Conversation

tomwhite
Copy link
Member

The idea here is to generalize blockwise slightly so you express a dependency of an output array block on input array blocks that you can't express using the index notation.

The example in the test is a simple form of merge_chunks where three input blocks are merged into one output block. (This will be useful in improving reduction, but that is not included in this PR.)

The reason we need this change is so that we can improve fusion optimizations. Currently merge_chunks uses the opaque map_direct function, which i) assumes that the array being merged has been materialized as a Zarr array, and ii) doesn't give any information about block mapping. By re-writing merge_chunks as a blockwise operation we get information about the block mapping that we will be able to use to fuse the merge_chunks with earlier operations in the DAG.

@tomwhite tomwhite mentioned this pull request Jan 17, 2024
20 tasks
@tomwhite tomwhite merged commit 49df637 into main Jan 18, 2024
7 checks passed
@tomwhite tomwhite deleted the generalize-blockwise branch January 18, 2024 12:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant